You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Creates coverage plots showing empirical coverage of posterior credible intervals.
A well-calibrated model would have coverage exactly match interval width (i.e. 95% credible interval contains the true value 95% of the time) as shown by a diagonal line.
The coverage for the provided simulated datasets is accompanied by credible intervals for coverage proportion (in form of a gray ribbon). These are calculated via the standard (conjugate) Beta-Binomial model for binomial proportions with a uniform prior.
Changes
adds module bayesflow.diagnostics.plots.coverage implementing coverage plot and coverage diff plot
adds auxiliary function compute_empirical_coverage in bayesflow.utils.plot_utils
Example
Here an example of a model that is "too uncertain", or too conservative. It over-covers:
Hi Lasse, I agree that this would be a very nice feature (which we implicitly care about when we compute the expected calibration error / ECE metric). And thank you for the PR, which I will review asap.
Hi Lasse, thanks for the PR and for following all contribution guidelines! I think the PR is almost merge-ready up to a small discussion on the following point:
Since the two new functions coverage and coverage_diff share a lot of code and logic, I wonder if it would make sense to merge them into a single function with an optional difference boolean flag, akin to what we have for the ECDF plots?
Also, do you happen to have a reference for the beta formulation of the coverage confidence bands which we can include in the docstring?
@stefanradev93 sure!
I agree with your first point, I merged the two as suggested. Regarding a source for the beta-binomial model I added a docstring reference to Chapter 2 of `Bayesian Data Analysis (2013, 3rd ed.) by Gelman A., et al. where this is covered.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #577.
Description
Creates coverage plots showing empirical coverage of posterior credible intervals.
A well-calibrated model would have coverage exactly match interval width (i.e. 95% credible interval contains the true value 95% of the time) as shown by a diagonal line.
The coverage for the provided simulated datasets is accompanied by credible intervals for coverage proportion (in form of a gray ribbon). These are calculated via the standard (conjugate) Beta-Binomial model for binomial proportions with a uniform prior.
Changes
bayesflow.diagnostics.plots.coverageimplementing coverage plot and coverage diff plotcompute_empirical_coverageinbayesflow.utils.plot_utilsExample
Here an example of a model that is "too uncertain", or too conservative. It over-covers: